Conversation
…plugins Fixes carvel-dev#1841 On Kubernetes 1.36+ the MutatingAdmissionPolicy admission plugin ships enabled by default. Its SharedInformer requires list+watch permission on mutatingadmissionpolicies and mutatingadmissionpolicybindings. If the deployed ClusterRole for the kapp-controller ServiceAccount lacks those permissions, the informer's cache never syncs, HasSynced() never returns true, and every mutating admission request into the aggregated apiserver fails permanently with 'not yet ready to handle request (reason: Forbidden)'. The existing serverResourceExists guard only confirms the API resource is present in cluster discovery, which is always true on k8s 1.36+ regardless of RBAC. This change extends the guard with a SelfSubjectAccessReview probe for both list and watch on the policy resource and its bindings resource before loading each admission-policy plugin. When the probe fails the plugin is skipped and an actionable warning is logged naming the missing (resource, verb) pair. The probe uses subjectaccessreviews/create, which is already granted in the shipped ClusterRole, no new RBAC required. Also applied to ValidatingAdmissionPolicy for symmetry; the same failure mode is possible on k8s 1.30+ if RBAC drifts. Signed-off-by: Neel Patel <neel5481@gmail.com>
neel5481
force-pushed
the
fix/1841-admission-policy-rbac-probe
branch
from
July 8, 2026 12:45
b8bac11 to
4b45307
Compare
joaopapereira
requested changes
Sep 23, 2026
joaopapereira
left a comment
joaopapereira
left a comment
Member
There was a problem hiding this comment.
I do not think that this approach is correct because in the end we will allow the user to surpass the RBAC configured. In my opinion we should go with something like:
- Fail readiness / don’t mark APIService Available when the resource exists in discovery but SSAR says no list/watch (or when the plugin is required and can’t sync).
- Or fail startup with a clear error: grant list/watch on mutatingadmissionpolicies / bindings (and validating equivalents), or set an explicit “disable admission-policy plugins” flag.
- Optional escape hatch: only skip if an explicit flag opt-out exists; default = refuse to serve writes.
I think that option 2 would be the safest approach for kapp-controller, but if we want to ensure that the controller at least is able to keep reconciling the AppCR 1 is a good approach.
Also we should ensure that in the release notes that we tell advise for CLusterRoles to be reapplied whenever deploying a new version of kapp-controller
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1841
On Kubernetes 1.36+ the MutatingAdmissionPolicy admission plugin ships enabled by default. Its SharedInformer requires list+watch permission on mutatingadmissionpolicies and mutatingadmissionpolicybindings. If the deployed ClusterRole for the kapp-controller ServiceAccount lacks those permissions, the informer's cache never syncs, HasSynced() never returns true, and every mutating admission request into the aggregated apiserver fails permanently with 'not yet ready to handle request (reason: Forbidden)'.
Signed-off-by: Neel Patel
What this PR does / why we need it:
The existing serverResourceExists guard only confirms the API resource is present in cluster discovery, which is always true on k8s 1.36+ regardless of RBAC. This change extends the guard with a SelfSubjectAccessReview probe for both list and watch on the policy resource and its bindings resource before loading each admission-policy plugin. When the probe fails the plugin is skipped and an actionable warning is logged naming the missing (resource, verb) pair.
The probe uses subjectaccessreviews/create, which is already granted in the shipped ClusterRole, no new RBAC required.
Also applied to ValidatingAdmissionPolicy for symmetry; the same failure mode is possible on k8s 1.30+ if RBAC drifts.
Which issue(s) this PR fixes:
Fixes #1841
Does this PR introduce a user-facing change?
Additional Notes for your reviewer:
Review Checklist:
a link to that PR
change
Additional documentation e.g., Proposal, usage docs, etc.: